<html>
<head><meta charset="utf-8"><title>Making Symbol more efficient: PR feedback? · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html">Making Symbol more efficient: PR feedback?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="217936149"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217936149" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217936149">(Nov 25 2020 at 20:36)</a>:</h4>
<p>Hey, folks! I've been working on understanding parallelism in Rust, and I noticed one opportunity for improvement. That's in how we handle interning strings as Symbols.  I decided to avoid placing all "static" symbols (symbols known at compile time) in the <code>Interner</code> table, and instead to just assign them static indices. </p>
<p>The advantage of this is that you can avoid taking a lock for resolution of static symbols. I measured, and for many Rust crates, the number of symbol lookups that can be resolved statically is between 45% and 75%. So, this could significantly reduce the amount of lock contention in <code>Symbol::intern</code>, if/when we ever get the compiler moved to parallel builds.</p>
<p>I have a branch where all of this works, and I've run <code>./x.py test src/tests/ui</code>, and everything passes. But there's still some cleanup left to do, so it's not ready for a real PR on rust-lang. Would anyone be interested in taking a look at this as it is now, and giving me feedback?</p>
<p>Tagging <span class="user-mention" data-user-id="239881">@Josh Triplett</span> , since I know you were interested in ||ism.</p>



<a name="217936288"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217936288" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217936288">(Nov 25 2020 at 20:37)</a>:</h4>
<p>The branch is: <a href="https://github.com/sivadeilra/rust/tree/user/ardavis/better_syms">https://github.com/sivadeilra/rust/tree/user/ardavis/better_syms</a></p>



<a name="217936492"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217936492" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217936492">(Nov 25 2020 at 20:39)</a>:</h4>
<p>The top of <code>compiler/rustc_macros/src/symbols.rs</code> has a bunch of comments that describe what's going on in this change.</p>



<a name="217937861"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217937861" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Hill <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217937861">(Nov 25 2020 at 20:52)</a>:</h4>
<p><span class="user-mention" data-user-id="274471">@Arlie Davis</span> If you open a draft PR, I can do a perf run</p>



<a name="217937896"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217937896" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Hill <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217937896">(Nov 25 2020 at 20:53)</a>:</h4>
<p>to see what effect it has so far</p>



<a name="217938430"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217938430" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217938430">(Nov 25 2020 at 20:57)</a>:</h4>
<p>Thanks, will do.</p>



<a name="217938521"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217938521" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217938521">(Nov 25 2020 at 20:58)</a>:</h4>
<p>As always, perf could actually be worse. I've done enough perf work to know how counter-intuitive it can be. :)</p>



<a name="217938703"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217938703" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217938703">(Nov 25 2020 at 21:00)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/pull/79425">https://github.com/rust-lang/rust/pull/79425</a> &lt;-- draft PR</p>



<a name="217939408"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217939408" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217939408">(Nov 25 2020 at 21:06)</a>:</h4>
<p><span class="user-group-mention" data-user-group-id="2943">@T-infra</span>: could someone look at <a href="https://github.com/rust-lang/rust/pull/79425/checks?check_run_id=1455727791">https://github.com/rust-lang/rust/pull/79425/checks?check_run_id=1455727791</a> ? That failure looks strange and spurious.</p>



<a name="217939480"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217939480" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217939480">(Nov 25 2020 at 21:06)</a>:</h4>
<p>(There was an issue with the PR, but "read-only filesystem" should not be the failure there.)</p>



<a name="217939485"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217939485" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217939485">(Nov 25 2020 at 21:06)</a>:</h4>
<p>It's probably because I omitted <code>Cargo.lock</code> from my PR. I just updated it to include the change to <code>Cargo.lock</code>.</p>



<a name="217939508"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217939508" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217939508">(Nov 25 2020 at 21:07)</a>:</h4>
<p>Sure, but the failure is still weird.</p>



<a name="217939534"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217939534" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217939534">(Nov 25 2020 at 21:07)</a>:</h4>
<p>Ah, ok.</p>



<a name="217939576"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217939576" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217939576">(Nov 25 2020 at 21:07)</a>:</h4>
<p>I added a test tool to the build, for counting the number of symbol lookups.  I'll probably remove it from the final for-reals PR.  It's interesting to be able to measure different crates, but probably not interesting enough to include in the repo.</p>



<a name="217940054"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217940054" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217940054">(Nov 25 2020 at 21:12)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> I <em>think</em> that's because the source directory is read-only on CI, and when there is no <code>Cargo.lock</code> cargo tries to write down a new one</p>



<a name="217940084"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217940084" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217940084">(Nov 25 2020 at 21:13)</a>:</h4>
<p>Agreed.</p>



<a name="217940107"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217940107" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217940107">(Nov 25 2020 at 21:13)</a>:</h4>
<p>Ah. So, weird failure message but legitimate failure?</p>



<a name="217940108"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217940108" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217940108">(Nov 25 2020 at 21:13)</a>:</h4>
<p>so that's expected</p>



<a name="217940119"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217940119" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217940119">(Nov 25 2020 at 21:13)</a>:</h4>
<p>yep</p>



<a name="217940132"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217940132" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217940132">(Nov 25 2020 at 21:13)</a>:</h4>
<p><span class="user-mention" data-user-id="121055">@Pietro Albini</span> Thank you.</p>



<a name="217940504"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217940504" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217940504">(Nov 25 2020 at 21:17)</a>:</h4>
<p>I'm expecting the perf results to be neutral, assuming we're not measuring parallel compiles right now. There will be some benefit for single-threaded compiles, but not much.</p>



<a name="217941995"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217941995" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217941995">(Nov 25 2020 at 21:31)</a>:</h4>
<p><span class="user-mention" data-user-id="274471">@Arlie Davis</span> You have some failures due to rustdoc issues.</p>



<a name="217942023"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217942023" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217942023">(Nov 25 2020 at 21:31)</a>:</h4>
<p>Will that prevent the perf run?</p>



<a name="217942461"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217942461" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217942461">(Nov 25 2020 at 21:36)</a>:</h4>
<p>If the try build fails, it would.</p>



<a name="217942465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217942465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217942465">(Nov 25 2020 at 21:36)</a>:</h4>
<p>So probably.</p>



<a name="217942539"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217942539" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217942539">(Nov 25 2020 at 21:37)</a>:</h4>
<p>Yeah. Fixed.</p>



<a name="217942652"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217942652" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217942652">(Nov 25 2020 at 21:39)</a>:</h4>
<p>Any thoughts on the PR itself?</p>



<a name="217942987"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217942987" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217942987">(Nov 25 2020 at 21:43)</a>:</h4>
<p>Reading...</p>



<a name="217943001"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943001" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943001">(Nov 25 2020 at 21:43)</a>:</h4>
<p>The comments at the top of <code>rustc_macros/src/symbols.rs</code> are the best place to start.</p>



<a name="217943011"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943011" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943011">(Nov 25 2020 at 21:43)</a>:</h4>
<p>That's what I'm reading.</p>



<a name="217943034"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943034" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943034">(Nov 25 2020 at 21:43)</a>:</h4>
<p>It looks like you've put a lot of effort into carefully figuring out the static symbols as efficiently as possible. The perf run will show if that pays off. I do wonder if it'd be possible to use a rustc-compile-time-generated <code>phf</code> perfect hash for that, to save yourself a lot of manual work.</p>



<a name="217943146"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943146" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943146">(Nov 25 2020 at 21:45)</a>:</h4>
<p>I also wonder if the fastest way to compare a 4-character symbol might just be to compare 32-bit numbers.</p>



<a name="217943180"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943180" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943180">(Nov 25 2020 at 21:45)</a>:</h4>
<p>Just got to your analysis about perfect hashes.</p>



<a name="217943234"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943234" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943234">(Nov 25 2020 at 21:46)</a>:</h4>
<p>Yeah, I thought of that. All sorts of fun strategies. The main thing, though, is just avoiding hitting the dynamic state _at all_, regardless of what static data structure you use.</p>



<a name="217943255"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943255" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943255">(Nov 25 2020 at 21:46)</a>:</h4>
<p>That much is completely understandable, yeah. Probably a major win for parallel.</p>



<a name="217943281"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943281" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943281">(Nov 25 2020 at 21:47)</a>:</h4>
<p>You've got another CI failure, this one looks easy to fix.</p>



<a name="217943712"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943712" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943712">(Nov 25 2020 at 21:52)</a>:</h4>
<p>The overall approach seems reasonable to me, and I like that you're doing as much as possible when rustc is compiled. If it's straightforward, I'd love to see a perf run comparing this approach to just using <code>phf</code> at build time in the simplest possible way. Whatever wins, let's use that.</p>



<a name="217943741"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943741" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943741">(Nov 25 2020 at 21:53)</a>:</h4>
<p>(If they roughly tie, maintaining less code would be nice. If your approach wins, by all means let's use it.)</p>



<a name="217943846"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943846" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943846">(Nov 25 2020 at 21:54)</a>:</h4>
<p>Yeah, it'll be interesting to see what the different metrics do.  Binary size, working set, CPU time, etc.  I suspect they're all in the noise, until we have substantial parallelism involved.</p>



<a name="217943885"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217943885" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217943885">(Nov 25 2020 at 21:55)</a>:</h4>
<p>I set up a Criterion-based test for just parsing files, but I don't have it using any parallelism yet.  I want to wire that up to rustc's rayon, and see what happens.</p>



<a name="217944552"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217944552" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217944552">(Nov 25 2020 at 22:03)</a>:</h4>
<p>If this change doesn't affect performance, you could try a draft PR on top that does parallelism by default, and do a perf run on that.</p>



<a name="217944562"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217944562" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217944562">(Nov 25 2020 at 22:03)</a>:</h4>
<p>(with and without this change, to see if it helps in the parallel case.)</p>



<a name="217944637"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217944637" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217944637">(Nov 25 2020 at 22:04)</a>:</h4>
<p>/me feels like there ought to be a way to ask for a perf run with parallelism, without having to make a commit manually enabling it.</p>



<a name="217944677"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217944677" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217944677">(Nov 25 2020 at 22:04)</a>:</h4>
<p>Thank you for working on this.</p>



<a name="217944681"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217944681" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217944681">(Nov 25 2020 at 22:04)</a>:</h4>
<p>Eh, I'll do the perf runs locally.  I just build a Threadripper (64 cores, 128 threads) just for compiler work.  :)</p>



<a name="217944716"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217944716" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217944716">(Nov 25 2020 at 22:05)</a>:</h4>
<p>Between the TLS changes, and changes like this, I'm really hoping that the bottlenecks in parallel compilation can go away and we can consider doing it by default.</p>



<a name="217944750"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217944750" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217944750">(Nov 25 2020 at 22:05)</a>:</h4>
<p>I sure hope so. Because we are so, so very slow right now.  Sigh.</p>



<a name="217945095"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217945095" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Notification Bot <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217945095">(Nov 25 2020 at 22:09)</a>:</h4>
<p>This topic was moved by <span class="user-mention silent" data-user-id="239881">Josh Triplett</span> to <a class="stream-topic" data-stream-id="122651" href="/#narrow/stream/122651-general/topic/Build.20times.20and.20build.20systems">#general &gt; Build times and build systems</a></p>



<a name="217946045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217946045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217946045">(Nov 25 2020 at 22:19)</a>:</h4>
<p>There's a new failure, related to marking tests as ignored.</p>



<a name="217946229"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217946229" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217946229">(Nov 25 2020 at 22:22)</a>:</h4>
<p>I think tidy shouldn't actually matter</p>



<a name="217946463"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217946463" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217946463">(Nov 25 2020 at 22:25)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> ^ -- CI doesn't need to pass for perf to run, we just need to be able to build things. Tidy, for example, or other tests don't matter.</p>



<a name="217946568"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/217946568" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#217946568">(Nov 25 2020 at 22:26)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> Ah, OK. I knew that not <em>everything</em> had to pass, but I didn't know which bits did or didn't well enough to want to waste bors time if I got it wrong.</p>



<a name="218339002"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218339002" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218339002">(Nov 30 2020 at 20:10)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span>  I have some perf results. Unsurprisingly, avoiding acquiring a mutex is a big win, in benchmarks that do nothing but look up symbols. I mean, we knew that would be the case, but it's still good to see the numbers. I wrote a basic benchmark using Criterion, which I've used a lot and which does a good job on rigorous analysis of data, warmups, etc. I did a run of rustc which printed out every symbol lookup, and then stored that in a fine. That's my test input; the benchmark just repeatedly looks up the same sequence of symbols (10,000 symbols or so), from the rustc source code.</p>
<p>Even for 1 CPU (no contention), static interning reduces lookup time by 13%, from 37ns to 31ns.<br>
For 2 to 4 CPUs, the improvement is above 21%.<br>
For 8 CPUs, the improvement is 49%.<br>
For 16 CPUs, 57%.<br>
For 24 CPUs, 58%.<br>
For 48 CPUs, 56%, [edited]<br>
For 64 CPUs, 58%.</p>
<p>So it looks like we reach a certain plateau. I'm guessing it's mainly because the test inputs are a mix of dynamic and static symbols. Every time we fail to find a symbol during static lookups, we acquire that mutex. There are ways to improve even that, of course, but I'll keep that out of scope for this work.</p>



<a name="218339274"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218339274" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218339274">(Nov 30 2020 at 20:13)</a>:</h4>
<p>That sounds awesome.</p>



<a name="218339337"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218339337" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218339337">(Nov 30 2020 at 20:13)</a>:</h4>
<p>Like all microbenchmarks, that doesn't translate to a significant win for overall parsing or compiling, of course.  But it does show that there is no cost to this, some minor gain, and it sets us up for parallelism.</p>



<a name="218339452"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218339452" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218339452">(Nov 30 2020 at 20:14)</a>:</h4>
<p>I'm looking forward to seeing the overall compiler perf results, too, especially for parallel operation.</p>



<a name="218339507"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218339507" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218339507">(Nov 30 2020 at 20:15)</a>:</h4>
<p>Does the plateau go up if you have a larger dataset to parallelize on?</p>



<a name="218339582"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218339582" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218339582">(Nov 30 2020 at 20:15)</a>:</h4>
<p>I'm guessing it's more about the ratio of static-to-dynamic symbols in that dataset, rather than the size of the set.  I could artificially add more static symbols, but I wanted a mix that reflected ordinary source code.</p>



<a name="218339740"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218339740" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218339740">(Nov 30 2020 at 20:16)</a>:</h4>
<p>That was what I was curious about: if you keep the ratio the same, but increase the dataset to one from a much much larger crate, how does it do?</p>



<a name="218339840"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218339840" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218339840">(Nov 30 2020 at 20:17)</a>:</h4>
<p>The dataset I'm testing is all of rustc's source code (across all crates). So, pretty big to begin with.</p>



<a name="218339964"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218339964" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218339964">(Nov 30 2020 at 20:18)</a>:</h4>
<p>Ah, interesting. That'll definitely be large, but it wouldn't surprise me if it has a different ratio of static to dynamic than regular code, in either direction.</p>



<a name="218339993"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218339993" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218339993">(Nov 30 2020 at 20:18)</a>:</h4>
<p>In any case, promising microbenchmark results.</p>



<a name="218340071"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340071" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340071">(Nov 30 2020 at 20:19)</a>:</h4>
<p>Are you planning to run the rustc-perf suite locally, or get it to run on the PR?</p>



<a name="218340215"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340215" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340215">(Nov 30 2020 at 20:20)</a>:</h4>
<p>I looked at the ratios for a bunch of common crates. Before my changes, most were in the range of 35% to 50% static symbols. Then I added the "common words" (a set of common identifiers, but not known to rustc's implementation), which raised the ratios to 50% to 75% or so.  I checked rust/compiler, rust/library, rust/src/toolsl, all of Servo, rand, syn, quote, a few others.</p>



<a name="218340278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340278">(Nov 30 2020 at 20:21)</a>:</h4>
<p>If the rustc-perf suite can be run locally, I'd like to start with that. Is it straightforward to run it?</p>



<a name="218340296"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340296" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340296">(Nov 30 2020 at 20:21)</a>:</h4>
<p><span class="user-mention silent" data-user-id="274471">Arlie Davis</span> <a href="#narrow/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F/near/218340215">said</a>:</p>
<blockquote>
<p>I looked at the ratios for a bunch of common crates. Before my changes, most were in the range of 35% to 50% static symbols. Then I added the "common words" (a set of common identifiers, but not known to rustc's implementation), which raised the ratios to 50% to 75% or so.  I checked rust/compiler, rust/library, rust/src/toolsl, all of Servo, rand, syn, quote, a few others.</p>
</blockquote>
<p>Nice analysis! That sounds like it should go into your PR's description.</p>



<a name="218340347"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340347" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340347">(Nov 30 2020 at 20:21)</a>:</h4>
<p><span class="user-mention silent" data-user-id="274471">Arlie Davis</span> <a href="#narrow/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F/near/218340278">said</a>:</p>
<blockquote>
<p>If the rustc-perf suite can be run locally, I'd like to start with that. Is it straightforward to run it?</p>
</blockquote>
<p>It's feasible. Not trivial, but feasible. Best place to ask is probably in <a class="stream" data-stream-id="247081" href="/#narrow/stream/247081-t-compiler.2Fperformance">#t-compiler/performance</a>.</p>



<a name="218340366"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340366" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340366">(Nov 30 2020 at 20:21)</a>:</h4>
<p>Many folks there know the details for how to run it locally.</p>



<a name="218340376"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340376" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340376">(Nov 30 2020 at 20:21)</a>:</h4>
<p>there's instructions in the readme: <a href="https://github.com/rust-lang/rustc-perf/tree/master/collector">https://github.com/rust-lang/rustc-perf/tree/master/collector</a></p>



<a name="218340492"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340492" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340492">(Nov 30 2020 at 20:22)</a>:</h4>
<p>Thanks.  I'm honestly not expecting much of a change at all, though, because currently other phases of compilation dominate. I've been using "compile libcore" as my main target, and in that case, most of the time goes to typeck.</p>



<a name="218340568"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340568" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340568">(Nov 30 2020 at 20:23)</a>:</h4>
<p>I would not trust benchmarks on the standard library tbh, it has lots of weird code</p>



<a name="218340595"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340595" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340595">(Nov 30 2020 at 20:23)</a>:</h4>
<p>like, no other codebase has that many lang_items</p>



<a name="218340677"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340677" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340677">(Nov 30 2020 at 20:24)</a>:</h4>
<p>Sure. But since the only benefit of this is reducing symbol lookup times, and the ratios between libcore and others are pretty similar, I expect similar results.</p>



<a name="218340714"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340714" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340714">(Nov 30 2020 at 20:24)</a>:</h4>
<p>Well, smaller heap space, too, but that's minor.</p>



<a name="218340719"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218340719" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218340719">(Nov 30 2020 at 20:24)</a>:</h4>
<p><span class="user-mention" data-user-id="274471">@Arlie Davis</span> What I'm personally hoping for is "neutral or marginally positive" with single-threaded rustc, and then significant improvements on parallel.</p>



<a name="218341009"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218341009" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218341009">(Nov 30 2020 at 20:26)</a>:</h4>
<p>(I'm also hoping those results stay roughly the same without the custom hashing.)</p>



<a name="218341051"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218341051" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218341051">(Nov 30 2020 at 20:27)</a>:</h4>
<p>Because you want to just use an ordinary hash table, or <code>phf</code>?</p>



<a name="218341263"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218341263" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218341263">(Nov 30 2020 at 20:28)</a>:</h4>
<p>Because optimizing the CPU-local symbol lookup seems entirely orthogonal to making it possible to run in parallel, and because <a href="https://github.com/rust-lang/rust/pull/79425#issuecomment-735244857">https://github.com/rust-lang/rust/pull/79425#issuecomment-735244857</a> . It'll be much easier to get this change in if it comes in two phases: localize symbol lookup, then potentially optimize symbol lookup.</p>



<a name="218341381"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218341381" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218341381">(Nov 30 2020 at 20:29)</a>:</h4>
<p>(I don't necessarily think this is going to need a <em>dozen</em> PRs, but I think it'll absolutely need at least 2.)</p>



<a name="218342847"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218342847" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218342847">(Nov 30 2020 at 20:40)</a>:</h4>
<p>Overall I agree. But I'm not eager to add <code>phf</code> as a dependency of <code>rustc</code>, at least not just for this. The fewer dependencies, the better.</p>



<a name="218343031"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218343031" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218343031">(Nov 30 2020 at 20:41)</a>:</h4>
<p>Yeah, dozen is a hyperbole, but there are clearly several orthogonal parts here.</p>



<a name="218343383"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218343383" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218343383">(Nov 30 2020 at 20:44)</a>:</h4>
<p>For example, I really wouldn't want to land the "divide and conquer" part here, unless there are significant performance benefits on non-synthetic benchmarks.</p>



<a name="218343475"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218343475" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218343475">(Nov 30 2020 at 20:45)</a>:</h4>
<p>The column-search part?  So, I can understand the reaction, but can you elaborate why?</p>



<a name="218343499"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218343499" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218343499">(Nov 30 2020 at 20:45)</a>:</h4>
<p>Which is not certain, because accesses to string representation of symbols are rare and we may be optimizing cold code.</p>



<a name="218343603"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218343603" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218343603">(Nov 30 2020 at 20:46)</a>:</h4>
<p><span class="user-mention silent" data-user-id="274471">Arlie Davis</span> <a href="#narrow/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F/near/218343475">said</a>:</p>
<blockquote>
<p>The column-search part?  So, I can understand the reaction, but can you elaborate why?</p>
</blockquote>
<p>Anything beyond a simple <code>HashMap</code>.<br>
Because complexity, no code is always better than code.</p>



<a name="218343616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218343616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218343616">(Nov 30 2020 at 20:46)</a>:</h4>
<p>The divide-and-conquer strategy only applies to the string --&gt; symbol translation, though.</p>



<a name="218343684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218343684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218343684">(Nov 30 2020 at 20:47)</a>:</h4>
<p>Agreed that no code is better than code.  But there's also "no dependency is better than dependency", and "no start-up heap allocs are better than start-up heap allocs".  It's trade-offs.</p>



<a name="218343845"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218343845" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218343845">(Nov 30 2020 at 20:48)</a>:</h4>
<p>I'm working on wiring up <code>phf</code> to this right now, so we'll see how it does.</p>



<a name="218343847"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218343847" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218343847">(Nov 30 2020 at 20:48)</a>:</h4>
<p>Sure, that's why perf needs to first show that the positive side of this trade-off exists at least.</p>



<a name="218344108"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218344108" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218344108">(Nov 30 2020 at 20:50)</a>:</h4>
<p><span class="user-mention" data-user-id="274471">@Arlie Davis</span> I assume you're doing the pre-compiled thing, so that at runtime the perfect hash is already compiled into the rustc binary?</p>



<a name="218344137"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218344137" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218344137">(Nov 30 2020 at 20:51)</a>:</h4>
<p>Also, you should probably start by trying a simple HashTable, and then show that phf is a win over that.</p>



<a name="218344772"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218344772" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218344772">(Nov 30 2020 at 20:56)</a>:</h4>
<p><span class="user-mention silent" data-user-id="274471">Arlie Davis</span> <a href="#narrow/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F/near/218343845">said</a>:</p>
<blockquote>
<p>I'm working on wiring up <code>phf</code> to this right now, so we'll see how it does.</p>
</blockquote>
<p>(If smarter string search strategy could be offloaded to a crate with simplicity of use similar to <code>HashMap</code>, it would certainly be great for rustc.)</p>



<a name="218345881"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218345881" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218345881">(Nov 30 2020 at 21:04)</a>:</h4>
<p>One interesting approach would be: Make <code>match</code> on string literals generate essentially the same optimal code, and then this code would just generate a single, trivial <code>match symbol_test { "foo" =&gt; ... }</code> expression.</p>



<a name="218346997"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218346997" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218346997">(Nov 30 2020 at 21:14)</a>:</h4>
<p><code>phf</code> is significantly slower. For single-threaded, PHF is 45% slower than column-divide.</p>



<a name="218347121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218347121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218347121">(Nov 30 2020 at 21:14)</a>:</h4>
<div class="codehilite"><pre><span></span><code>intern/1                time:   [47.577 ns 47.685 ns 47.801 ns]
                        change: [+40.669% +45.576% +49.536%] (p = 0.00 &lt; 0.05)
                        Performance has regressed.
</code></pre></div>



<a name="218347237"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218347237" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218347237">(Nov 30 2020 at 21:15)</a>:</h4>
<p>As number of threads go up, the effect decreases, because lock contention becomes the dominant factor.</p>



<a name="218737682"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218737682" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218737682">(Dec 03 2020 at 18:51)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span>  Hey, an update. I did more rigorous perf investigations, and found that <code>phf</code> actually has the best perf.  I've updated the PR, removed all of my "work-in-progress" stuff, switched it to use <code>phf</code>, and converted the PR draft to a regular PR.  <a href="https://github.com/rust-lang/rust/pull/79425">https://github.com/rust-lang/rust/pull/79425</a></p>



<a name="218737859"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218737859" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218737859">(Dec 03 2020 at 18:52)</a>:</h4>
<p>I had thought that <code>phf</code> was slower, when really what happened was that <code>phf</code> lookups were completing quickly enough that they enabled more thread contention, because the current thread would move to the next symbol quickly enough. Anyway, the details aren't terribly important. The key thing is that <code>phf</code> is good enough, and fits the model that I want, where the static lookup table is truly static (no heap allocs, no initialization work).</p>



<a name="218754255"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754255" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754255">(Dec 03 2020 at 21:04)</a>:</h4>
<p><span class="user-mention" data-user-id="274471">@Arlie Davis</span> How hard might it be to add a <code>-Z</code> option to track and list stats for the most looked-up dynamic symbols, with an eye towards adding them to the static table?</p>



<a name="218754324"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754324" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754324">(Dec 03 2020 at 21:05)</a>:</h4>
<p>...that also raises an interesting idea.</p>



<a name="218754386"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754386" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754386">(Dec 03 2020 at 21:05)</a>:</h4>
<p>Pretty straight-forward, although I think I would want that to be something that can be conditionally-enabled, so that it doesn't affect normal compiles.</p>



<a name="218754411"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754411" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754411">(Dec 03 2020 at 21:05)</a>:</h4>
<p>Fair point; might not be possible to make it zero cost when not in use.</p>



<a name="218754440"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754440" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754440">(Dec 03 2020 at 21:05)</a>:</h4>
<p>I have a little standalone binary that links to rustc_span and rustc_session. It crawls a directory tree, finds all *.rs files, parses them, and dumps symbol stats.</p>



<a name="218754447"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754447" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754447">(Dec 03 2020 at 21:06)</a>:</h4>
<p>(I wish Rust had a "static branch" mechanism like the Linux kernel, for such things.)</p>



<a name="218754609"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754609" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754609">(Dec 03 2020 at 21:07)</a>:</h4>
<p>I don't know if this is possible or not, but:<br>
Suppose, in addition to the compile-time table, we reserved a large range of ID space for additional static IDs, and provided the ability to load a single pre-compiled phf table from a disk file to use for those static IDs?</p>



<a name="218754637"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754637" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754637">(Dec 03 2020 at 21:07)</a>:</h4>
<p>Then, if you have a huge codebase, you could create a cached set of symbols.</p>



<a name="218754648"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754648" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754648">(Dec 03 2020 at 21:07)</a>:</h4>
<p>rustc could even help you handle that, in the incremental case.</p>



<a name="218754720"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754720" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754720">(Dec 03 2020 at 21:08)</a>:</h4>
<p>I spent a bunch of time this morning looking at the perf impact. Right now, the perf impact is precisely zero, because everything else is so slow. I've compared rustc with and without my symbols changes, and all of the results are in the noise.  I've compiled with the <code>parallel_compiler</code> option enabled, used <code>-Zthreads=NN</code>, and seen that it really is using multiple threads.  But the time that we spend parsing is small, compared to analysis.</p>



<a name="218754747"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754747" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754747">(Dec 03 2020 at 21:08)</a>:</h4>
<p>And on your <em>next</em> compile after you have that cache, you might have 90+% static lookups, because you cached all your common symbols. If the cache is outdated, no big deal.</p>



<a name="218754779"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754779" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754779">(Dec 03 2020 at 21:08)</a>:</h4>
<p>(Yeah, all of what I'm mentioning there only matters if symbol lookup turns out to be a bottleneck still.)</p>



<a name="218754811"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754811" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754811">(Dec 03 2020 at 21:09)</a>:</h4>
<p>For libcore, CPU time to compile is about 28 seconds.  Time for lexing, parsing, building AST, and stopping before analysis is about 5.4 seconds.</p>



<a name="218754819"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754819" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754819">(Dec 03 2020 at 21:09)</a>:</h4>
<p>The impact is in the noise even on your Threadripper?</p>



<a name="218754843"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754843" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754843">(Dec 03 2020 at 21:09)</a>:</h4>
<p>And, during lexing / parsing, we're currently single-threaded, even if we're using -Zthreads=64.</p>



<a name="218754874"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218754874" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218754874">(Dec 03 2020 at 21:10)</a>:</h4>
<p>-Zthreads appears to enable multithreading during analysis, but not during lexing / parsing.</p>



<a name="218755019"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218755019" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218755019">(Dec 03 2020 at 21:11)</a>:</h4>
<p>lexing, parsing, and macro expansion/resolution are all going to be single threaded right now, yes</p>



<a name="218755036"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218755036" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218755036">(Dec 03 2020 at 21:11)</a>:</h4>
<p>We _could_ do some multi-threaded parsing, by doing things like parsing separate files.  Whenever we load <code>mod foo;</code>, we could start that, then later wait for all of the async / parallel parsing to complete.</p>



<a name="218755091"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218755091" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218755091">(Dec 03 2020 at 21:11)</a>:</h4>
<p>Or, we could pipeline some of it, where lexing was done by a separate thread, which fed results to the thread consuming it. But that seems unlikely to gain much.</p>



<a name="218755239"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218755239" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218755239">(Dec 03 2020 at 21:12)</a>:</h4>
<p>I've been looking at applying the static-symbol technique to Rust Analyzer, since RA spends far more of its time in lexing/parsing than rustc does. It might help more there, in the near term.</p>



<a name="218756654"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218756654" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218756654">(Dec 03 2020 at 21:25)</a>:</h4>
<p>rust-analyzer is already shockingly fast :)</p>



<a name="218756851"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218756851" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218756851">(Dec 03 2020 at 21:26)</a>:</h4>
<p>Ehhh, it could be faster. And crash less. There are a number of projects that I work with, where I have to disable RA just to get a usable editor again.</p>



<a name="218756901"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Making%20Symbol%20more%20efficient%3A%20PR%20feedback%3F/near/218756901" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Arlie Davis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Making.20Symbol.20more.20efficient.3A.20PR.20feedback.3F.html#218756901">(Dec 03 2020 at 21:27)</a>:</h4>
<p>It's mostly great, and I love the design.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>